WebToData Tutorial

Due to the number of different ways that data can be stored in Html files, WebToData provides a number of powerful options for extracting data from web pages. The WebToData Tutorial will assist you to understand and know when to use the different data extraction utilities.

It is recommended that you review each of the following sections to gain an understanding of how the different data extraction features work. Once you have a broad understanding of how data extraction works you may want to revisit specific sections as necessary.

To learn more about the different data extraction tools select one of the options below.

Data Sensing The automatic data sensing tool.  Always use as the first option.

User Defined Basic user defined data searching.

Pattern Matching Complex user defined data searching using Pattern Matching.

When using the Data extraction tools you must first select the required tool from the Data menu.

Then select the Transfer Web Page to Data command from the Actions menu.


Data Sensing

Data Sensing should be used as the default method for transferring a web page to a data file. If the results are not as expected then the User Defined or Pattern Matching methods should be used.  Even if you know that Data Sensing will not work, you will still need to run this tool to find out the settings for the other tools.

The following table represents the format of a table often found in web pages.  It is a simple representation but the key points remain valid regardless of the complexity of the presentation.

The key point to note is that each row in the table would represent a complete record in a database. Each row refers to a unique set of data.

Code 	First Name 	Second Name 	Address 	Post Code 	Phone

001	Fred 	Smith 	Wood St 	7089 	7000000

002 	Bob 	Wong 	Oak St 	7099 	8000000

003 	Melissa 	Hicky 	Walnut St 	7088 	9000000

004 	Jasmine 	Nott 	Pine St 	7022 	1000000

005 	Ann 	Smith 	Wood St 	7089 	7000000

006 	Jessica 	Wong 	Oak St 	7099 	8000000

007 	Carley 	Evans 	Blackbutt St 	7011 	2000000

008 	John 	Albert 	Pine St 	7022 	3000000

009 	Erin 	Jarvis 	Hoop St 	7033 	400000

[Next]   [Previous]


Data Sensing Output - Formatted Text
Data Sensing Output – Formatted Text

The Formatted Text view shows the end result of all processing completed by WebToData.  When using WebToData for data extraction, the final database ready file will be shown in this window. The example below shows the end result of processing the html file, using Data Sensing, in the Formatted Text window.

To get the example data as shown in the screen shot above it is important to ensure that the Use Data Sensing option is selected.  This menu determines which data extraction tool is used when processing the web file.  A tick will appear next to the selected option.

To process the html file:

1.	Go to the Actions menu and choose Transfer Web Page to Data.

You can save any window at any time.

[Next]   [Previous]


Data Sensing Output - Text File
Data Sensing Output

To gain a sense of how Data Sensing works and how WebToData processes web pages it is worthwhile to view the results of the data processing in the Text File window. Viewing data in this window will also help you understand what the data looks like before final processing.  In some cases there will be a major difference between the data found in this window and the data found in the Formatted Text window. By comparing the results between the Text File window and the Formatted Text window it is possible to select the appropriate data extraction tool for the task at hand.  Check information on the other tools for further information.

If you wish you could save the data found in this window.

[Home]  [Previous]


Saving Data

To save this file as a data file.

1.	Go to the File menu and choose Save.  It is important that the text file type is selected when saving files.


User Defined Data Extraction

The following HTML table shows a table that will be incorrectly processed by the Data Sensing tool.  It is the same data used in the first Data Sensing example but it is presented differently on the web page. Before explaining why it is handled incorrectly it is important to see what it would look like if Data Sensing were used.

[Next]   [Previous]


User Defined Data Extraction - Incorrect
User Defined Data Extraction – Incorrect Data

The following table shows how WebToData incorrectly processes the previous table if the Data Sensing tool is used. You will note that the final output is a row of two commas. It can be a confusing problem, but it does have a simple solution.

[Next]  [Previous]


User Defined Data Extraction - Text File View
User Defined Data Extraction – Text File View

The following view shows how the Text File window presents the same data that was displayed incorrectly in the Formatted Text window.

By looking at this view we are able to establish that WebToData has found the data but the Data Sensing tool is not selecting the correct data. It is picking up the double commas as the predominate feature in this database. This easily fixed by using the User Defined Data selection tool.

Before moving on to the next section I want you to note two things.

Firstly note how all the data for a record is on one line. This is important for understanding when to use User Defined Data settings. For the User Defined Data setting to work all data must be on the one line.

Secondly count how many commas appear in a line of data from the sample above.

I get six.

[Next]   [Previous]


User Defined Data Extraction - Setting Up
User Defined Data Extraction – Setting Up

To setup WebToData for User Defined Data extraction you must tell the program how many commas appear in the text with the data in it.  That is why you were asked to count the number of commas before!

To open the Window shown below choose Program Options from the Options menu and go to the User Defined Data section.

Once this window is open then:

1/	In the Delimiters box insert the number of delimiters.  In this case six.

2/	Ensure that the User Defined Data Delimiters are being used. It is selected when a tick is evident.

3/	Choose to remember settings if desired.  This should be selected if a number of the similar web pages are to be processed.

4/	Press the OK button to continue.

Your settings should be the same as the example below.

[Next]   [Previous]


User Defined Data Extraction - Formatted View
User Defined Data Extraction – Formatted View

The following sample demonstrates how WebToData will process the data if the correct user settings are applied in the User Defined Data options.  It also shows that User Defined Data should be selected in the Data menu.

The final data file should be saved.

[Home]   [Previous]


Pattern Matching

The following table shows a table that will be processed incorrectly by both the Data Sensing and User Defined Data Options.  By analysing the table it is evident that the record for the database is spread over a number of lines. Fortunately WebToData has the ability to find data in this type of web page and rebuild it into a database file.

Code 	First Name	Second Name 	Address

Post Code 	Phone

001 	Fred 	Smith 	Wood St

7089 	7000000

002 	Bob 	Wong 	Oak St

7099	8000000

 003 	Melissa 	Hicky 	Walnut St

7088 	9000000

Before moving on to how to set up WebToData to find and extract this type of data we need to look at how the Data Sensing tool handles it.

[Next]   [Previous]


Pattern Matching - Incorrect
Pattern Matching – Incorrect Output

The following sample shows how the previous table is handled if the Data Sensing tool is used.  It is evident that this tool has only found part of the database record.  We need something else to find the data.

But before moving on it is still important to run the Data Sensing tool. If we do not run the Data Sensing tool we will not be able to see what the data will like in the Text File view.

[Next]   [Previous]


Pattern Matching - Text File View
Pattern Matching – Text File View

The following sample Text File window demonstrates how the data will look after running the Data Sensing tool. It shows disjointed data that is basically useless for any database.  But WebToData can rebuild this type of file.

Before moving on lets pause to examine why Pattern Matching is the only option that will work.

It is simple really.  Data Sensing and User Defined Data options will not work because of the way the record is split over more than one line. That is all you have to look for when deciding whether or not to use Pattern Matching.

[Next]   [Previous]


Pattern Matching - Setting Up Pattern Matching Options
Pattern Matching – Setting Up Pattern Matching Options

Pattern Matching options are available by choosing Pattern Matching Options from the Options menu.

It is a straight forward process to set up Pattern Matching options.

1/	We have to define how many lines make up one section of the pattern. If we go back to the previous section we can count 5 lines to make up one section. Specifically if we go to the line starting with “Code” as shown in the previous sample and move to the last line in the first pattern section, starting with “Post Code”, there are 5 lines.

2/	In the Number of Lines to Process box, 5 is inserted.

3/	Now we investigate the first line in the first pattern section. We find that there are 4 commas or delimiters in the line.  There is text in the line and this line has data in it that is required for our database.

4/	In the first line section we add the information found in our investigation.

5/	For the second line we determine that it is a blank line and record this information in the Line 2 part of the Pattern Matching Options.  We also indicate that it is not required for our database by not selecting to Add

6/	For the third line in the pattern we find that it has one comma.  We then indicate that there is one delimiter for line 3 and we have no need to add this line to the database.

7/	We find that the fourth line is blank and record this line accordingly.

8/	In the fifth line we find that there are 3 commas and text that needs to be added to the database.  This information is recorded for the fifth line.

9/	Select Pattern Matching so it is on.

The following example shows how the above settings will look in the Pattern Matching section of the Options Form.

[Next]   [Previous]


Pattern Matching - Formatted Text
Pattern Matching – Formatted Text

If the Pattern Matching Options are managed correctly the output of the example file will be as displayed below.

The file can be saved as with the other tools.

[Home]   [Previous]